Virtual overlay system and method for occluded objects

ABSTRACT

The present invention provides a system and method of displaying a representation of a road sign. The method comprises receiving information associated with the road sign, the information associated with the road sign comprising a front face of the road sign and a location of the road sign in a road system; receiving an image from a camera, the road sign being within the field of view of the camera; determining, using the image, an amount of the road sign that is obstructed; generating, in dependence on the amount obstructed, the representation of the road sign using the information associated with the road sign; and outputting the representation of the road sign for display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a 371 of International Application No.PCT/EP2017/057008, filed Mar. 23, 2017, which claims priority to GBPatent Application 1605137.7, filed Mar. 25, 2016, the contents of bothof which are incorporated by reference herein in their entirety.

TECHNICAL FIELD

The present disclosure relates to a virtual overlay system and methodfor occluded objects and particularly, but not exclusively, to a systemand method for overlaying a virtual road sign for a vehicle driver.Aspects of the invention relate to a method of displaying arepresentation of a road sign, to a non-transitory computer-readablestorage medium, to a system for displaying a representation of a roadsign, to a processor and to a vehicle.

BACKGROUND

Vehicle drivers often rely on road signs that provide information,warning and/or instructions such as speed limits, directions, roadlayout diagrams, height restrictions etc. However, overgrown trees andbushes can sometimes occlude and/or obscure road signs, impeding orpreventing vehicle drivers from reading the road signs.

A further scenario that may prevent road signs from being read isdiscussed with reference to FIGS. 1(a) and (b). FIG. 1(a) shows a roadsystem 100 comprising two parallel traffic lanes (a first lane 102 a anda second lane 102 b) and a road sign 104. Vehicles in the two trafficlanes travel in the same direction. A vehicle 106 is shown travelling inthe road system 100 in the second lane 102 b. A driver of the vehicle106 has a field of view depicted by the lines 108. The road sign 104 isahead of the vehicle 106 and is in the field of view 108 of the driver.

FIG. 1(b) shows the road system 100 with the vehicle 106 and a goodsvehicle 110. The goods vehicle 110 is shown travelling parallel to thevehicle 106 in the first lane 102 a. The presence of the goods vehicle110 affects the field of view of the vehicle driver, shown as lines 112.In particular, the goods vehicle blocks driver of the vehicle 106 fromreading the road sign 104.

In the scenario of FIG. 1(b), the road sign 104 may be visible to thedriver of the vehicle 106 prior to the road sign 104 being blocked fromview by the goods vehicle 110. However, this may be for an insufficientlength of time for the driver to read the road sign 104.

The inability for drivers to read road signs to obtain pertinentinformation presents a safety hazard, for example, if a driver of a tallvehicle is unaware of a height restriction of a low bridge.Additionally, some drivers may attempt to mitigate the problem of a roadsign being blocked by a parallel vehicle by altering their vehicle speed(slowing down or speeding up) in order to maintain a clear view of theroad sign. However, this inconsistent driving speed may cause a hazardto other road users.

The present invention has been devised to mitigate or overcome at leastsome of the above-mentioned problems.

SUMMARY OF THE INVENTION

Aspects and embodiments of the invention provide a method, anon-transitory computer-readable storage medium, a system, a processorand a vehicle as claimed in the appended claims.

According to an aspect of the present invention there is provided amethod of displaying a representation of a road sign, the methodcomprising: receiving information associated with the road sign, theinformation associated with the road sign comprising a front face of theroad sign and a location of the road sign in a road system; receiving animage from a camera, the road sign being within the field of view of thecamera; determining, using the image, an amount of the road sign that isobstructed; generating, in dependence on the amount obstructed, therepresentation of the road sign using the information associated withthe road sign; and outputting the representation of the road sign fordisplay.

An advantage of the method is that if a road sign is partly or whollyobstructed from the view of the camera, a representation of the roadsign is outputted for display, for example to a user, enabling the userto read the road sign even though it is partly or wholly obstructed tothe user.

In the image received by the camera, the road sign may be directlyviewable by the camera in the field of view, or the road sign may bewithin the field of view but be partially or fully obscured in the imagereceived by the camera.

The information associated with the road sign may be received from thecamera, and the representation is generated based on the image and thelocation from the camera. The location of the road sign may convenientlybe determined from the images received from the camera.

Optionally, the information associated with the road sign is stored in amemory, and the representation is generated based in dependence on theinformation in the memory.

Optionally, the information associated with the road sign is receivedfrom a remote server and stored in a database, and the representation isgenerated based in dependence on the information in the database. Abenefit of this is if the road sign is never visible to the camera, itcan still be displayed. Where the information associated with the roadsign is received from a remote server, the location of the vehicle mayconveniently be determined from a location determining means (such as aGPS unit) within the vehicle. The location of the vehicle may bedetermined using the location determining means and, accordingly, theroad signs that should be in the field of view of the camera.

Optionally, the method comprises outputting the representation of theroad sign to a head-up display.

Optionally, the method comprises outputting the representation of theroad sign on the head-up-display to be overlaid over the position of theroad sign when viewed by a user.

Optionally, the camera comprises a wide angle lens.

A non-transitory computer-readable storage medium may comprisecomputer-readable instructions for a computer processor to carry out themethod above.

According to another aspect of the present invention there is provided asystem for displaying a representation of a road sign, the systemcomprising: an input arranged to receive: information associated withthe road sign, the information associated with the road sign comprisinga front face of the road sign and a location of the road sign in a roadsystem; and an image from a camera, the road sign being within the fieldof view of the camera; a processor arranged to: determine, using theimage, an amount of the road sign that is obstructed; generate, independence on the amount obstructed, the representation of the road signusing the information associated with the road sign; and an outputarranged to output the representation of the road sign for display.

Optionally, the information associated with the road sign is receivedfrom the camera, and the representation is generated based on the imageand the location from the camera. The location of the road sign mayconveniently be determined from the images received from the camera.

Optionally, the information associated with the road sign is receivedfrom a remote server and stored in a database, and the representation isarranged to be generated based on the information in the database. Wherethe information associated with the road sign is received from a remoteserver, the location of the vehicle may conveniently be determined froma location determining means (such as a GPS unit) within the vehicle.The location of the vehicle may be determined using the locationdetermining means and, accordingly, the road signs that should be in thefield of view of the camera.

Optionally, the representation of the road sign is arranged to be outputto a head-up-display.

Optionally, the representation of the road sign on the head-up-displayis arranged to be output to a head-up-display to be overlaid over theposition of the road sign when viewed by a user.

Optionally, the camera comprises a wide angle lens.

According to another aspect of the present invention there is provided aprocessor configured to: receiving information associated with the roadsign, the information associated with the road sign comprising a frontface of the road sign and a location of the road sign in a road system;receiving an image from a camera, the road sign being within the fieldof view of the camera; determining, using the image, an amount of theroad sign that is obstructed; generating, in dependence on the amountobstructed, the representation of the road sign using the informationassociated with the road sign; and outputting the representation of theroad sign for display.

A vehicle may comprise the system described above and/or the processordescribed above.

According to another aspect of the invention there is provided a methodof displaying a representation of a road sign, the method comprising:receiving, at an overlay system, information associated with the roadsign, the information associated with the road sign comprising a frontface of the road sign and a location of the road sign in a road system;receiving, at a camera input module of the overlay system, an image froma camera, the road sign being within the field of view of the camera;determining, in a sign detection module of the overlay system, using theimage, an amount of the road sign that is obstructed; generating, in anocclusion detection module of the overlay system, in dependence on theamount obstructed, the representation of the road sign using theinformation associated with the road sign; and outputting from theoverlay system to a display the representation of the road sign fordisplay.

The overlay system may comprise an electronic processor having anelectrical input for receiving signals comprising the informationassociated with the road sign and the image from the camera. The overpaysystem may further comprise an electronic memory device electricallycoupled to the electronic processor and having instructions storedtherein. The determining, generating and outputting steps may comprisethe processor being configured to access the memory device and executeinstructions stored therein such that it is operable to carry out thesteps of the method of displaying a representation of a road sign.

The sign detection module, occlusion detection module and camera inputmodule may comprise a plurality of processing arrangements within theprocessor. Alternatively the sign detection module, occlusion detectionmodule and camera input module may each be located in their ownprocessor such that the overlay system comprises a plurality ofprocessors each of which is coupled to the electronic memory device.

Within the scope of this application it is expressly intended that thevarious aspects, embodiments, examples and alternatives set out in thepreceding paragraphs, in the claims and/or in the following descriptionand drawings, and in particular the individual features thereof, may betaken independently or in any combination. That is, all embodimentsand/or features of any embodiment can be combined in any way and/orcombination, unless such features are incompatible. The applicantreserves the right to change any originally filed claim or file any newclaim accordingly, including the right to amend any originally filedclaim to depend from and/or incorporate any feature of any other claimalthough not originally claimed in that manner.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1(a) and (b) have already been described above by way ofbackground, in which:

FIG. 1(a) is a schematic top view of a road system with a road sign anda vehicle; and

FIG. 1(b) is a schematic top view of the road system of FIG. 1(a) with agoods vehicle travelling parallel to the vehicle.

One or more embodiments of the invention will now be described, by wayof example only, with reference to the accompanying drawings, in which:

FIG. 2(a) is a schematic top view of a road system with a road sign anda vehicle comprising an overlay system according to an embodiment of thepresent invention;

FIG. 2(b) is a schematic front view of the road sign as visible to theoverlay system of FIG. 2(a);

FIG. 3(a) is a schematic top view of the road system of FIG. 2(a) with agoods vehicle travelling parallel to the vehicle;

FIG. 3(b) is a schematic front view of the road sign as visible to theoverlay system of FIG. 3(a);

FIG. 3(c) is a schematic front view of the road sign with a partialoverlay;

FIG. 4(a) is a schematic top view of the road system of FIG. 2(a) with agoods vehicle travelling parallel to the vehicle;

FIG. 4(b) is a schematic front view of the road sign as visible to theoverlay system of FIG. 4(a);

FIG. 4(c) is a schematic front view of the road sign with a fulloverlay;

FIG. 5 is a flowchart of a process carried out by the overlay system ofFIG. 2;

FIG. 6 is a schematic block diagram of the overlay system of FIG. 2;

FIG. 7 is a schematic block diagram of an overlay system according to anembodiment of the present invention; and

FIG. 8 is a flowchart of a process carried out by the overlay system ofFIG. 7.

DETAILED DESCRIPTION

FIG. 2(a) shows a road system 200 comprising two parallel traffic lanes(a first lane 202 a and a second lane 202 b) and a road sign 204.Vehicles in the two traffic lanes travel in the same direction. Avehicle 206 is shown travelling in the road system 200 in the secondlane 202 b. The vehicle 206 comprises a camera 208, an overlay system210 and a display 212. In other embodiments the road sign may be locatedon a gantry over the lanes 202 a, 202 b.

The camera 208 and the display 212 are each operatively connected to theoverlay system 210. The display 212 is a head-up display. In otherembodiments, the camera 208 and/or display 212 are wirelessly connectedto the overlay system 210.

The camera 208 has a field of view depicted by the lines 214, which issubstantially the same as a field of view as the driver. The road sign204 is ahead of the vehicle 206 and is in the field of view 214 of thecamera 208. FIG. 2(b) shows a front face 216 of the road sign 204 thatis visible to the camera 208. The front face 216 provides informationregarding an upcoming road layout of the road system 200. The whole ofthe front face 216 is visible to the camera 208, and hence also to thedriver of the vehicle 206.

The overlay system 210 is arranged to continuously monitor the roadsystem 200 using the camera 208 to detect road signs. If any road signs(e.g. road sign 204) are detected, then information associated with theroad signs are stored in the sign database 256 (as shown in FIG. 5). Theinformation associated with the road signs includes the front face 216of the road sign and the location of the road sign in a road system 200.If any of the stored road signs are subsequently blocked from view ofthe camera, the overlay system 210 is arranged to determine the extentof the blockage (i.e. how much of the road sign is not visible),retrieve the stored road sign, and output the blocked portion to thedisplay 212.

FIG. 3(a) shows the road system 200 with the vehicle 206 and a goodsvehicle 220. The goods vehicle 220 is shown travelling parallel to thevehicle 206 in the first lane 202 a. The presence of the goods vehicle220 affects the field of view of the camera 208, shown as lines 222. Inparticular, the goods vehicle 220 blocks part of the front face 216 ofthe road sign 204 such that only a portion 224 of the road sign isvisible to the camera 208.

FIG. 3(b) shows the portion 224 of the front face 216 of the road sign204 that would be visible to the vehicle driver due to the obstructionby the side of the goods vehicle 220. As the field of view of the driveris substantially the same at the camera 208, the visible portion 224 issubstantially the same. However, in the present embodiment, the overlaysystem 210 determines that the road sign 204 would be partiallyobstructed by the good vehicle 220 from the view of the driver, and inresponse, the overlay system 210 outputs the obstructed portion of theroad sign to the display 212.

FIG. 3(c) shows a view of the vehicle driver through the heads-updisplay with the obstructed portion 226 of the road sign being displayedto complete the front face 216 of the road sign 204.

FIG. 4(a) shows the road system 200 with the vehicle 206 and a goodsvehicle 230. The goods vehicle 230 is shown travelling parallel to thevehicle 206 in the first lane 202 a. The presence of the goods vehicle230 affects the field of view of the camera 208, shown as lines 232. Inparticular, the goods vehicle 230 blocks the whole front face 216 of theroad sign 204 such that none of the road sign is visible to the camera208.

FIG. 4(b) shows the view that would be visible to the vehicle driver dueto the obstruction by the good vehicle 230. As the field of view of thedriver is substantially the same at the camera 208, the road sign iscompletely obstructed from the view of the driver. However, in thepresent embodiment, the overlay system 210 determines that the road sign204 would be completely obstructed by the good vehicle 410 from the viewof the driver, and in response, the overlay system 210 outputs theobstructed portion of the road sign to the display 212.

FIG. 4(c) shows a view of the vehicle driver through the heads-updisplay with the obstructed road sign 234 displayed to complete thefront face 216 of the road sign 204.

FIG. 5 shows an embodiment of the overlay system 210 in greater detail.The overlay system 210 comprises a controller 250, a camera input module252, a sign detection module 254, a memory 256, an occlusion detectionmodule 258 and an overlay output module 260. The camera input module252, the sign detection module 254, the memory 256, the occlusiondetection module 258 and the overlay output module 260 are eachoperatively connected to the controller 250.

The camera input module 252 is arranged to receive input from the camera208. The camera 208 is arranged to capture images of the road system 200at between 24 to 240 frames per second. The camera 208 may be astereoscopic camera.

The sign detection module 254 is arranged to perform object recognitionanalysis on the images received by the camera 208 to determine thepresence of road signs in the images. For example, the sign detectionmodule 254 may be configured to detect road signs by: shape (e.g.circular, triangular or rectangular); predominant colour (e.g. yellow,green, blue or brown); and/or border colour (e.g. red or white).

The sign detection module 254 is further arranged to determine thelocation and orientation of any detected road signs relative to thevehicle 206.

In other embodiment the camera 208 is arranged to perform thefunctionality of the sign detection module 254. In such embodiment, thecamera 208 is arranged to perform object recognition analysis on theimage captured to determine the presence of road signs in the images.The camera 208 is further arranged to determine the location and/ororientation of any detected road signs relative to the vehicle 206. Thecamera 208 output comprises one or more of: the image data, the locationdata and the orientation data.

Any road signs detected in the road system 200, for example road sign204, are cropped from the camera image and stored in the memory 256. Thecropped image comprises the front face of the road sign 204. The roadsigns are removed from the memory 256 after a predetermined length oftime or when the storage space for the memory 256 is full. Arepresentation of the front face of the road sign 204 may be createdfrom the cropped image of the road sign 204. Then the memory may store arepresentation of the front face of the road sign 204, the locationand/or orientation data in association with the road sign 204.

The occlusion detection module 258 is arranged to compare sequentialimages from the camera 208 which have been identified as comprising aroad sign in order to detect whether less of the road sign is visible insubsequent images.

The overlay output module 260 is arranged to determine and outputrepresentations of signs stored in the memory 256 to the display 212.The overlay output module 260 may be arranged to output therepresentation based on the determined location and orientation of theroad sign being displayed.

FIG. 6 shows a process 300 carried out by an embodiment of the overlaysystem 210. The process 300 begins with the camera input module 252receiving at Step 302 an image (i.e. a video frame) from the camera 208.Then the sign detection module 254 detects at Step 304 any road signspresent in the camera image, as well as the locations of any road signsrelative to the vehicle 206.

The sign detection module 254 checks at Step 306 whether any road signswere detected in the image. If one or more road signs are detected, theocclusion detection module 258 checks at Step 308 whether road signswere detected in the previous image received from the camera 208. If noroad signs were previously detected, then the sign detection module 254extracts (i.e. crops) any road signs from the current image and storesat Step 310 any extracted road signs in the memory 256. Following this,the process 300 returns to Step 302.

If following the check at Step 306 whether any road signs were detectedin the image, no road signs are detected, then the occlusion detectionmodule 258 checks at Step 312 whether road signs were detected in theprevious image received from the camera 208. The occlusion detectionmodule 258 takes into account the movement of the vehicle 206 (forwardsand lateral) and the locations of any road signs relative to the vehicle206 determined at Step 304, in order to determine whether the vehicle206 has moved past the location of the sign. In other words, theocclusion detection module 258 checks whether the road sign is no longerexpected to be in view of the camera 208.

If following the check at Step 312, no road signs were previouslydetected, the process 300 returns to Step 302. However, if following thecheck at Step 312 road signs were previously detected and the vehicle206 has not travelled past the road sign, this means that the road signhas now been obstructed from view of the camera 208. Accordingly, theoverlay output module 260 retrieves at Step 314 the previously detectedroad sign from the memory 256. Then the overlay output module 260determines the size, orientation and location of where the sign isexpected to be. The overlay output module 260 also determines arepresentation of the road sign corresponding to the expected size,orientation and location of the road sign, and outputs at Step 316 therepresentation of the road sign for display on the display 212.Following this, the process 300 returns to Step 302. The effect of thisis that, as the vehicle 206 approaches the location of the road sign204, the representation of the road sign increases in size.

Returning to the check at Step 308, if the outcome of the check is thatone or more road signs were detected in the previous image received fromthe camera 208, then the occlusion detection module 258 determines atStep 318 whether the full front face of the road sign is visible, or ifonly part of the road sign is visible. The occlusion detection module258 takes into account the movement of the vehicle 206 (forwards andlateral) and the locations of any road signs relative to the vehicle 206determined at Step 304, in order to determine whether the vehicle 206has begun to move past the location of the sign. In other words, theocclusion detection module 258 checks whether the full front face of theroad sign is no longer expected to be in view of the camera 208.

If following the check of Step 318, the full front face of the road signis visible, then there is no occlusion or obstruction, and the process300 returns to Step 302.

However, if following the check of Step 318 only a part of the road signis visible, this is indicative that the road sign is being partiallyblocked from the field of view of the camera 208. Accordingly, theoverlay output module 260 retrieves at Step 320 the previously detectedroad sign from the memory 256. Then the overlay output module 260determines the size, portion, orientation and location of where the signis expected to be. The overlay output module 260 also determines arepresentation of the road sign corresponding to the expected size,portion, orientation and location of the road sign, and outputs at Step322 the representation of the road sign for display on the display 212.Following this, the process 300 returns to Step 302.

In an alternative embodiment, an overlay system comprises locationdetermining means (e.g. a GPS receiver) and a database storinginformation associated with a set of road signs (the informationassociated with the road signs includes the front face 216 of the roadsign and the location of the road sign in a road system 200). Theoverlay system is arranged to determine the location of the vehicleusing the location determining means and, accordingly, the road signsthat should be in the field of view of the camera 208. The overlaysystem is arranged to determine if the road signs that should be in thefield of view of the camera 208 are blocked, and outputs arepresentation of the road sign in response.

FIG. 7 shows the alternative embodiment of the overlay system 400. Theoverlay system 400 comprises a controller 402, a camera input module404, a sign detection module 406, a sign database 408, an overlay outputmodule 410, a GPS receiver 412, an occlusion detection module 414, and agaze detection module 416. The camera input module 404, the signdetection module 406, the sign database 408, the overlay output module410, the GPS receiver 412, the occlusion detection module 414, and thegaze detection module 416 are each operatively connected to thecontroller 402.

The camera input module 404 is arranged to receive input from the camera208. The vehicle 206 further comprises an internal camera 418 connectedto the camera input module 404. The internal camera 418 is directed atthe driver of the vehicle, and may be focussed on the driver's face.

The sign database 408 comprises information associated with a set ofroad signs (the information associated with the road signs includes thefront face 216 of the road sign and the location of the road sign in aroad system 200). The set of road signs may be all the road signs in ageographic area, or the road signs along a route along which the vehicleis travelling. As shown in FIG. 7 the sign database 408 is locatedlocally on the vehicle 206. The sign database may receive informationassociated with road signs from a remote server. In an alternativearrangement the overlay system 400 may be in communication with a remotedatabase 408. For example, the remote database 408 may be located inanother vehicle or cloud or any structure/building near the road. If thevehicle is on a guided route then the information associated with a setof road signs in relation to the guided route take priority in thecommunication with a remote database 408.

The GPS receiver 412 is arranged to determine the location of thevehicle 206 by receiving timing information from GPS satellites as isknown in the art. In other embodiments, the location of the vehicle maybe determined by other means such as cellular network multilateration asis known in the art.

The sign detection module 406 is substantially the same as the signdetection module 254 and is arranged to perform object recognitionanalysis on the images received by the camera 208 to determine thepresence of road signs in the images. The sign detection module 406 isfurther arranged to determine the location of any detected road signsrelative to the vehicle 206 using information from the GPS receiver 412to determine road signs that are expected to be in the field of view ofthe camera 208.

The occlusion detection module 414 is substantially the same as theocclusion detection module 258 and is arranged to compare sequentialimages from the camera 208 which have been identified as comprising aroad sign in order to detect whether less of the road sign is visible insubsequent images.

The overlay output module 410 is substantially the same as the overlayoutput module 260 and is arranged to determine and outputrepresentations of signs stored in the sign database 408 to the display212.

The gaze detection module 416 is arranged to determine where the driveris looking, and what the driver is looking at. For example, taking thefollowing 3D coordinates:

Object Viewed by Driver (e.g. a road sign): O=[x_(O), y_(O), z_(O)]

Driver's Head: D=[x_(D), y_(D), z_(D)]

Forward Facing Camera: C=[x_(C), y_(C), z_(C)]

And the following vectors:

DO→=[O−D]

Determine eye gaze vector for the driver via the internal camera 418based on eye gaze direction and head position.

DC→=[D−C]

Both the Driver's Head and the camera 208 are in the same coordinatesystem.

CO→=[O−C]

This vector shows the path between the camera 208 and the object viewedby the Driver. DO→ is outputted by the internal camera 418 and thecamera 208 is able to position this in its 3D model.

To determine the distance to the object, the internal camera 418 useseye vergence to provide an estimate of distance. The camera 208 isstereoscopic to estimate target distance and is used for closest matchto eye vergence. The object detected by the camera 208 that isintercepted or nearest to the gaze vector is considered the object ofvisual interest.

If there is a head movement of the driver, then the internal camera 418recalculates the new co-ordinates of the eye position (since the camera208 and the internal camera 418 are of fixed positions). As the vehiclemoves, the relative position of the object also changes. The gazedetection module 416 tracks the new coordinates as the vehicle moves.The gaze detection module 416 can calculate the spatial co-ordinates ofthe object (e.g. road sign), which is intercepted by the gaze vector.

FIG. 8 shows a process 450 carried out by the overlay system 400. Theprocess 450 begins with the GPS receiver 412 receiving at Step 452 thelocation of the vehicle 206. Based on the received location, the signdetection module 406 determines at Step 454, using information from thesign database 408, whether there are any signs that are expected to bein the field of view of the camera 208. If not, the process 450 returnsto Step 452.

If following the check of Step 454, there are expected to be road signsin the field of view of the camera 208, then the camera input module 404receives at Step 456 an image (i.e. a video frame) from the camera 208.The occlusion detection module 414 determines at Step 458 whether theroad signs that are expected to be in the field of view of the camera208 are visible. The occlusion detection module 414 checks at Step 460whether the full front face of the road sign is visible, or if none oronly part of the road sign is visible. If the full front face of theroad sign is visible, then the process 450 returns to Step 452.

However, if following the check of Step 460, none or only part of theroad sign is visible to the camera 208, then the overlay output module410 retrieves at Step 462 the road sign from the sign database 408. Thenthe overlay output module 410 determines the size, portion, orientationand location of where the sign is expected to be. The overlay outputmodule 410 also determines a representation of the road signcorresponding to the expected size, portion, orientation and location ofthe road sign, and outputs at Step 464 the representation of the roadsign for display on the display 212. Following this, the process 450returns to Step 452.

It will be appreciated that the functionality of camera input module252; 404, the sign detection module 254; 406, the overlay output module260; 410, the occlusion detection module 258; 414 and the gaze detectionmodule 416 may be performed by one or more processors. When performed bymultiple processors the processors may be located independently fromeach other.

It will be appreciated that road sign comprises static signs and activesigns having changeable content where the content may be changed overtime. Active signs may be car parking information, road work and trafficinformation signs.

Many modifications may be made to the above examples without departingfrom the scope of the present invention as defined in the accompanyingclaims.

For example, if a road sign is fully obscured, then the representationof the road sign on the display 212 may be at any orientation andlocation, and not necessarily match the actual orientation and locationof the road sign.

In other embodiments, the display 212 may be a display in a centreconsole of the vehicle or a display in the dashboard of the vehicle.

Optical character recognition may be used to detect words and placenames from the signs.

The invention claimed is:
 1. A computer-implemented method of displayinga representation of a road sign, the method comprising: receivinginformation associated with the road sign, the information associatedwith the road sign comprising a front face of the road sign and alocation of the road sign in a road system; receiving an image from acamera of a vehicle, the road sign being within the field of view of thecamera and the image being one of a sequence of images from the camera;determining, using the image, an amount of the road sign that isobstructed comparing sequential images from the camera; generating,based on the amount of the road sign that is obstructed, therepresentation of the road sign using the information associated withthe road sign; and outputting the representation of the road sign fordisplay on a display of the vehicle.
 2. The computer-implemented methodof claim 1, wherein the information associated with the road sign isreceived from the camera and the representation is generated based onthe image and the location from the camera.
 3. The computer-implementedmethod of claim 1, wherein the information associated with the road signis received from a remote server and stored in a database, and therepresentation is generated based on the information in the database. 4.The computer-implemented method of claim 1, wherein the display is ahead-up display.
 5. The computer-implemented method of claim 4,comprising outputting the representation of the road sign on thehead-up-display to be overlaid over the position of the road sign whenviewed by a user.
 6. The computer-implemented method of claim 1, whereinthe camera comprises a wide angle lens.
 7. The computer-implementedmethod of claim 1, wherein the information associated with the road signis determined by the sign detection module and the representation isgenerated based on the determined information, the sign detection moduleis further arranged to determine the location and orientation of anydetected road signs relative to the vehicle.
 8. A non-transitorycomputer-readable storage medium comprising computer-readableinstructions for a computer processor to carry out thecomputer-implemented method of claim
 1. 9. A virtual overlay system fordisplaying a representation of a road sign, within a vehicle, the systemcomprising: an input arranged to receive: information associated withthe road sign, the information associated with the road sign comprisinga front face of the road sign and a location of the road sign in a roadsystem; and an image from a camera of the vehicle, the road sign beingwithin the field of view of the camera and the image being one of asequence of images from the camera; a processor arranged to: determine,using the image, an amount of the road sign that is obstructed 11 ycomparing sequential images from the camera; and generate, based on theamount of the road sign that is obstructed, the representation of theroad sign using the information associated with the road sign; and anoutput arranged to output the representation of the road sign fordisplay on a display of the vehicle.
 10. The system of claim 9, whereinthe information associated with the road sign is received from thecamera, and the representation is generated based on the image and thelocation from the camera.
 11. The system of claim 9, wherein theinformation associated with the road sign is received from a remoteserver and stored in a database, and the representation is arranged tobe generated based on the information in the database.
 12. The system ofclaim 9, wherein the display is a head-up-display.
 13. The system ofclaim 12, wherein the representation of the road sign is arranged to beoutput to the head-up-display to be overlaid over the position of theroad sign when viewed by a user of the system.
 14. The system of claim9, wherein the camera comprises a wide angle lens.
 15. The system ofclaim 9, wherein the information associated with the road sign isdetermined by the sign detection module and the representation isgenerated based on the determined information, the sign detection moduleis further arranged to determine the location and orientation of anydetected road signs relative to the vehicle.
 16. A vehicle comprisingthe system of claim
 9. 17. An electronic processor comprising: anelectrical input for receiving signals comprising: informationassociated with the road sign, the information associated with the roadsign comprising a front face of the road sign and a location of the roadsign in a road system; and image information from a camera of a vehicle,the road sign being within the field of view of the camera and the imageinformation being one of a sequence of images from the camera; and anelectronic memory device electrically coupled to the electronicprocessor and having instructions stored therein; wherein the processoris configured to: access the memory device and execute instructionsstored therein such that it is operable to determine, using the imageinformation, an amount of the road sign that is obstructed by comparingsequential images from the camera; generate, based on the amountobstructed, the representation of the road sign using the informationassociated with the road sign; and output the representation of the roadsign for display on a display of the vehicle.
 18. A vehicle comprisingthe electronic processor of claim 17.